Kernel Spectral Clustering and applications

In this chapter we review the main literature related to kernel spectral clustering (KSC), an approach to clustering cast within a kernel-based optimization setting. KSC represents a least-squares support vector machine based formulation of spectral clustering described by a weighted kernel PCA objective. Just as in the classifier case, the binary clustering model is expressed by a hyperplane in a high dimensional space induced by a kernel. In addition, the multi-way clustering can be obtained by combining a set of binary decision functions via an Error Correcting Output Codes (ECOC) encoding scheme. Because of its model-based nature, the KSC method encompasses three main steps: training, validation, testing. In the validation stage model selection is performed to obtain tuning parameters, like the number of clusters present in the data. This is a major advantage compared to classical spectral clustering where the determination of the clustering parameters is unclear and relies on heuristics. Once a KSC model is trained on a small subset of the entire data, it is able to generalize well to unseen test points. Beyond the basic formulation, sparse KSC algorithms based on the Incomplete Cholesky Decomposition (ICD) and $L_0$, $L_1, L_0 + L_1$, Group Lasso regularization are reviewed. In that respect, we show how it is possible to handle large scale data. Also, two possible ways to perform hierarchical clustering and a soft clustering method are presented. Finally, real-world applications such as image segmentation, power load time-series clustering, document clustering and big data learning are considered.Comment: chapter contribution to the book "Unsupervised Learning Algorithms

Solving a large dense linear system by adaptive cross approximation

An efficient algorithm for the direct solution of a linear system associated with the discretization of boundary integral equations with oscillatory kernels (in two dimensions) is described without having to compute the complete matrix of the linear system. This algorithm is based on the unitary-weight representation, for which a new construction based on adaptive cross approximation is proposed. This low rank approximation uses only a small part of the entries to construct the adaptive cross representation, and therefor the linear system can be solved efficiently.nrpages: 22status: publishe

Sparse spectral clustering method based on the incomplete Cholesky decomposition

A new sparse spectral clustering method using linear algebra techniques is proposed. This method exploits the structure of the Laplacian to construct its approximation, not in terms of a low rank approximation but in terms of capturing the structure of the matrix. The approximation is based on the incomplete Cholesky decomposition with an adapted stopping criterion, it selects a sparse data set which is a good representation of the full data set. With this approximation the eigenvalue problem can be reduced to a smaller problem. To obtain the indicator vectors from the eigenvectors the method proposed by [Zha et al., Spectral relaxation for k-means clustering ] is adapted, which computes a pivoted LQ factorization of the eigenvector matrix. This formulation gives also the possibility to extend the method to out-of-sample points.nrpages: 20status: publishe

On the fast reduction of symmetric rationally generated Toeplitz matrices to tridiagonal form

In this paper two fast algorithms that use orthogonal similarity transformations to convert a symmetric rationally generated Toeplitz matrix to tridiagonal form are developed, as a means of finding the eigenvalues of the matrix efficiently. The reduction algorithms achieve cost efficiency by exploiting the rank structure of the input Toeplitz matrix. The proposed algorithms differ in the choice of the generator set for the rank structure of the input Toeplitz matrix.nrpages: 22status: publishe

Transforming a hierarchical into a unitary-weight representation

In this paper we consider a class of hierarchically rank structured matrices, including some of the hierarchical matrices occurring in the literature, such as hierarchically semiseparable (HSS) and certain H2-matrices. We describe a fast O(r3n log(n)) and stable algorithm to transform this hierarchical representation into a so-called unitary-weight representation, as introduced in an earlier work of the authors. This reduction allows the use of fast and stable unitary-weight routines (or by the same means, fast and stable routines for sequentially semiseparable (SSS) and quasiseparable representations, used by other authors in the literature), leading e.g. to direct methods for linear system solution and for the computation of all the eigenvalues of the given hierarchically rank structured matrix.nrpages: 27status: publishe

Implicit double shift QR-algorithm for companion matrices

In this paper an implicit (double) shifted QR-method for computing the eigenvalues of companion and fellow matrices will be presented. Companion and fellow matrices are Hessenberg matrices, that can be decomposed into the sum of a unitary and a rank 1 matrix. The Hessenberg, the unitary as well as the rank 1 structures are preserved under a step of the QR-method. This makes these matrices suitable for the design of a fast QR-method. Several techniques already exist for performing a QR-step. The implementation of these methods is highly dependent on the representation used. Unfortunately for most of the methods compression is needed since one is not able to maintain all three, unitary, Hessenberg and rank 1 structures. In this manuscript an implicit algorithm will be designed for performing a step of the QR-method on the companion or fellow matrix based on a new representation consisting of Givens transformations. Moreover, no compression is needed as the specific representation of the involved matrices is maintained. Finally, also a double shift version of the implicit method is presented